IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


DEVELOPMENT OF VISUALIZATION OF LEARNING 
OUTCOMES USING CURRICULUM MAPPING 


Takashi Ikuta and Yasushi Gotoh 

Niigata University 

8050, Ikarashi 2no-cho, Niigata City, Niigata, Japan 


ABSTRACT 

Niigata University has started to develop the Niigata University Bachelor Assessment System (NBAS). The objective is 
to have groups of teachers belonging to educational programs discuss whether visualized learning outcomes are 
comprehensible. Discussions based on teachers' subjective judgments showed in general that visualized learning 
outcomes express students' natural abilities and, unlike simple GPA and grade data, are conducive to grasping the level of 
success in reaching attainment targets. In future, it will be necessary to examine amendments to weighting and lessons 
and assessments that correspond to weighting. 
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1. INTRODUCTION 

Amid the increasingly fast-paced development of science and technology and the globalization of economic 
activities, how to assure graduate attributes has become a matter of the utmost importance. As graduate 
attribute frameworks, Japan has “graduate power” and debates about quality assurance in different fields on 
the U.K. QAA model. Our purpose, in terms of such quality assurance, is to make the basic knowledge and 
understanding which students must acquire as their grounding in a relevant discipline; the domain-specific 
skills; the generic skills; and the learning methods and grade assessments needed to obtain such knowledge 
and skills, reference points in forming educational curricula in different fields. At present, trials are underway 
in the fields of management, languages, literature, law, biology, home economics, mechanical engineering 
and mathematical science. 

In parallel to this type of discussion about quality assurance in different fields, it will also be necessary 
for teachers' groups to discuss individual educational programs. As Kawashima (2008) points out, teachers' 
groups involved in educational programs must discuss learning outcomes with the aim of fostering human 
resources. If a teacher is not able to perceive learning outcomes as being related to the actual learning 
outcomes of her own students, it is highly likely that quality assurance in different fields will end up as an 
abstract concept. 

At the same time, there are a number of difficulties in the visualization of learning outcomes. In PDP 
(Personal Development Planning) advocated in the U.S., students themselves perform reflection on the basis 
of the results transcript provided by the university and the learning process record known as PDR ( Personal 
Development Records) compiled by the student. This is supposed to foster students' ability to plan their own 
lifelong development, but it has also been reported that PDP is time-consuming and expensive and hinders a 
proper grasp of learning outcomes (Benesse). What is needed is a system which enables students to visualize 
learning outcomes for attainment targets. 

Niigata University has started to develop the Niigata University Bachelor Assessment System (NBAS) on 
the basis of these considerations. This system sets lower attainment targets in four educational target 
domains, namely: knowledge and understanding, domain-specific skills, generic skills and attitude, and 
visualizes these by displaying on a radar chart students' level of attainment for the corresponding learning 
outcomes. By visualizing learning outcomes as a radar’ chart display, it becomes possible to discuss students' 
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quality assurance from the point of view of generic skills and attitude, as well as from the point of view of 
knowledge and understanding and domain-specific skills. 

Ikuta and Gotoh (2011) proposed PDCA by teachers' groups on the basis of the relative importance of 
educational target domains in educational programs. Teachers' groups have a bird's eye view of the 
curriculum as a whole, but if quality assurance of graduate attributes is not to end up as an abstract concept 
without any substance, they need to examine concrete learning outcomes visualized for each student. More 
specifically, what they must do is verify whether visualized learning outcomes are comprehensible, i.e. 
whether the radar charts displayed by the system reflect students' attributes and skills. 


2. OBJECTIVE 

The objective is to have groups of teachers belonging to educational programs discuss whether visualized 
learning outcomes are comprehensible. 


3. METHOD 

Teaching staff from the life sciences, forest environment studies, and agricultural engineering programs and 
teachers from the Institute of Education and Student Affairs took part in the study. The study started in 
October 2010 and is scheduled to end in June 2012, i.e., it is still ongoing. 

The visualization method for learning outcomes is the one shown by Ikuta and Gotoh (2011). First, the 
educational targets domain was split into domain-specific academic knowledge, domain-specific skills, 
generic skills and attitude, and attainment targets were set at the lower level of this educational target domain. 
Next, a contribution ratio was assigned to the attainment targets in each subject forming an educational 
program. For example, in subject A, the attainment target in knowledge and understanding was 50%, 
domain-specific skills 30%, generic skills 10% and attitude 10%. In the educational target domain, several 
attainment targets were set. For example, where there were contributions to several attainment targets in 
knowledge and understanding, a further 50% was allocated to each attainment target. A curriculum map 
drawn up in this way is called a “weighted curriculum map.” In providing such weighting, lessons and 
assessments must be carried out by a corresponding method. 

Feaming outcome was the total score obtained by multiplying a student's grade assessment by this 
contribution ratio and the number of credits. To give an example, if a student scored 80 points in subject A, 
his score for knowledge and understanding was obtained by multiplying 80 (points) x 0.5 (50%) x 2 (number 
of credits) to achieve a score of 80. This score was totalled for each attainment target to obtain the learning 
outcome. Weighting was performed from October 2010 to March 2011. 

To find out whether intuitive comprehension of learning outcomes visualized on the basis of actual 
students' results was possible, students' performance data up to March 2012 (data at the end of the third year 
of the life sciences program, and data at the end of the fourth year of the forest environments studies and 
agricultural engineering programs) were used in order to draw up a radar chart. At the same time, materials 
were also prepared for understanding a student's relative position on each attainment target. 

Using these data, teachers belonging to the programs were interviewed and the results analyzed. The 
analysis transformed speech data into textual data, split these data for each segment and then categorized 
them. Although three educational programs were examined in this study, Niigata University offers 42 
programs and in future the data must be supplemented and verified. For this reason MAXQDA10 was used 
as the analysis software. 


4. RESULTS 

4.1 Visualized Learning Outcomes are Comprehensible 

In general, visualized learning outcomes clearly express students' particular characteristics and are 
comprehensible. In forest environment studies, the nature of the curriculum means that the subjects studied 
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by students are more or less identical and in many cases the shape of the radar chart display was similar. The 
difference between learning outcomes for students with good grades and students with poor grades was also 
immediately obvious from the size of the radar chart. (Figure. 1) 


Knowledge end understanding. Domain-specific skills. 





2008 ( 1 ) 
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2009 ( 1 ) 

2009 ( 2 ) 
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Figure 1 . An example of visualized learning outcomes 

By looking at the shape of the radar chart for each student, it is possible to understand which specialized 
domain subject students were learning and when. Students with an extremely low GPA have a distinctive 
radar chart display and since they can grasp visually which attainment targets have not been reached teachers 
are likely to find such radar charts useful when advising students. 

In life sciences, there are four specialized domains: housing studies, clothing studies, dietary habits and 
family resource management and since the subjects taken by students in each of these domains are virtually 
identical, typical patterns were generally confirmed for all four. 

It was proposed that using visualized learning outcomes in this way to carry out peer reviews among 
students in the same domain might be useful in setting targets for the next stage. 


4.2 Need for Pointers for Self-Assessment of Learning Outcomes 

In order to enable students to self-assess their own learning outcomes, some teachers thought that they should 
be provided with some kind of indicator or pointer. The idea is that for example, if one role model was drawn 
up for forest environment studies and four role models for life sciences in the domains of housing studies, 
clothing studies, dietary habits and family resource management, respectively, and if students could compare 
themselves to these role models, self-assessment would become easier. 

There are a number of methods concerning which pointers to use. Suggestions included: using actual data 
from excellent students and virtual data; showing the minimum line; comparing one's own performance with 
the standard performance by using the average value and the mode in subjects taken by that student; or 
combining these methods, i.e. comparing one’s own position to both the excellent line and the minimum line. 


4.3 Need for Continuous Examination of Curriculum Map Weighting is 
Needed 


When the relative position of students for each attainment target was discussed, attention was drawn to the 
fact that some attainment targets did not seem right. Some students who had a feeling that in terms of 
attainment targets they should be placed higher up, found that they were not particularly highly placed. 
When the weighted curriculum map was checked, it was found that only a small number of subjects had been 
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weighted for that attainment target and students who scored poorly had done so because they had not taken a 
weighted subject. 

On this basis, it seemed clear that further debate was needed about whether attainment targets were too 
fragmented or whether there were too few weighted subjects. 

4.4 Need for Discussion about Whether Lesson Contents and Assessments 
Correspond to Weighting 

Where teachers' groups in life sciences programs weighted curriculum maps by giving, for example, 50% to 
knowledge and understanding and 30% to domain-specific skills, they shared a recognition that lessons and 
assessments must also be carried out by corresponding methods. As a result, compared to pre -weighting 
(around March 2011), at present (March 2012), teachers are more aware of the relationship between 
attainment targets and lessons and assessments and revision is possible. In addition, attention was also drawn 
to the fact that teachers had greater awareness of the connection between other subjects making up the 
educational program and their own subject. 


5. DISCUSSION 

Discussions based on teachers' subjective judgments showed in general that visualized learning outcomes 
express students' natural abilities and, unlike simple GPA and grade data, are conducive to grasping the level 
of success in reaching attainment targets. In future, it will be necessary to examine amendments to weighting 
and lessons and assessments that correspond to weighting. To identify the reliability of visualized learning 
outcomes, some kind of direct assessment method such as learning portfolio or performance assessment will 
be need. 

Ikuta and Gotoh (201 1) proposed that “If individual teachers revise the syllabus, extract attainment targets 
and consider assessment methods and grade allocations in the educational program as a whole, they will be 
able to discuss educational programs for fostering human resources in general not only from the point of 
view of domain-specific knowledge, but also from the point of view of generic skills and attitude. Such 
discussions will have wide-ranging potential application in assuring quality in higher education.” 

Among teachers belonging to educational programs, there are disagreements even about individual 
generic skills and discussions may finish in abstract definitions. We want individual teachers to say, “What 
do these generic skills mean in concrete terms for students in my class? What lessons should I design to 
foster such skills? How should I assess them?” For this reason, as in the present study, it is essential to 
continue to examine how learning outcomes are being visualized for real, individual students and to provide 
weighting to the whole curriculum by taking a panoramic view of it. 
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